{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature engineering\n",
    "\n",
    "[Feature Engineering for Time Series Forecasting](https://www.trainindata.com/p/feature-engineering-for-forecasting)\n",
    "\n",
    "\n",
    "- In this notebook, we will create features suitable to predict the CO concentration in the next hour. \n",
    "\n",
    "- We assume that we have data available up to the hour before the forecast.\n",
    "\n",
    "<img src='../images/forecasting_framework.png' width=\"600\" height=\"600\">\n",
    "\n",
    "We want to predict the pollutant concentration at time t, and we know the concentration up to t-1. So for each t, we can use data up to t-1. \n",
    "\n",
    "Except for the timestamp, because we know at which time we want to predict pollutants.\n",
    "\n",
    "Let's create some features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from feature_engine.creation import CyclicalFeatures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function summarizes the various steps in\n",
    "# the previous notebook.\n",
    "\n",
    "def load_data():\n",
    "\n",
    "    # Data lives here.\n",
    "    filename = \"../datasets/AirQualityUCI_ready.csv\"\n",
    "\n",
    "    # Load data: only the time variable and CO.\n",
    "    data = pd.read_csv(\n",
    "        filename,\n",
    "        usecols=[\"Date_Time\", \"CO_sensor\", \"RH\"],\n",
    "        parse_dates=[\"Date_Time\"],\n",
    "        index_col=[\"Date_Time\"],\n",
    "    )\n",
    "\n",
    "    # Sanity: sort index.\n",
    "    data.sort_index(inplace=True)\n",
    "\n",
    "    # Reduce data span.\n",
    "    data = data.loc[\"2004-04-01\":\"2005-04-30\"]\n",
    "\n",
    "    # Remove outliers\n",
    "    data = data.loc[(data[\"CO_sensor\"] >= 0) & (data[\"RH\"] >= 0)]\n",
    "\n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>RH</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>1224.0</td>\n",
       "      <td>56.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1215.0</td>\n",
       "      <td>59.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1115.0</td>\n",
       "      <td>62.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1124.0</td>\n",
       "      <td>65.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1028.0</td>\n",
       "      <td>65.3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor    RH\n",
       "Date_Time                           \n",
       "2004-04-04 00:00:00     1224.0  56.5\n",
       "2004-04-04 01:00:00     1215.0  59.2\n",
       "2004-04-04 02:00:00     1115.0  62.4\n",
       "2004-04-04 03:00:00     1124.0  65.0\n",
       "2004-04-04 04:00:00     1028.0  65.3"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load data.\n",
    "\n",
    "data = load_data()\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- timestamp in the index. \n",
    "\n",
    "- CO_sensor: carbon monoxide concentration.\n",
    "\n",
    "- RH: relative humidity (in the air)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract time related features\n",
    "\n",
    "These are features that capture information from the timestamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>RH</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Day</th>\n",
       "      <th>Day_of_week</th>\n",
       "      <th>Hour</th>\n",
       "      <th>is_weekend</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>1224.0</td>\n",
       "      <td>56.5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1215.0</td>\n",
       "      <td>59.2</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1115.0</td>\n",
       "      <td>62.4</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1124.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1028.0</td>\n",
       "      <td>65.3</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor    RH  Month  Week  Day  Day_of_week  Hour  \\\n",
       "Date_Time                                                                   \n",
       "2004-04-04 00:00:00     1224.0  56.5      4    14    4            6     0   \n",
       "2004-04-04 01:00:00     1215.0  59.2      4    14    4            6     1   \n",
       "2004-04-04 02:00:00     1115.0  62.4      4    14    4            6     2   \n",
       "2004-04-04 03:00:00     1124.0  65.0      4    14    4            6     3   \n",
       "2004-04-04 04:00:00     1028.0  65.3      4    14    4            6     4   \n",
       "\n",
       "                     is_weekend  \n",
       "Date_Time                        \n",
       "2004-04-04 00:00:00           1  \n",
       "2004-04-04 01:00:00           1  \n",
       "2004-04-04 02:00:00           1  \n",
       "2004-04-04 03:00:00           1  \n",
       "2004-04-04 04:00:00           1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract date and time features.\n",
    "\n",
    "data[\"Month\"] = data.index.month\n",
    "data[\"Week\"] = data.index.isocalendar().week\n",
    "data[\"Day\"] = data.index.day\n",
    "data[\"Day_of_week\"] = data.index.day_of_week\n",
    "data[\"Hour\"] = data.index.hour\n",
    "\n",
    "# find out if it is a weekend.\n",
    "data[\"is_weekend\"] = np.where(data[\"Day_of_week\"]>4, 1, 0)\n",
    "\n",
    "# Show new variables\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lag features\n",
    "\n",
    "Lag features are past values of the variable that we can use to predict future values.\n",
    "\n",
    "<img src='../images/lag_features.png' width=\"600\" height=\"600\">\n",
    "\n",
    "\n",
    "I will use the following lag features to predict the next hour's pollutant concentration:\n",
    "\n",
    "- The pollutant concentration for the previous hour (t-1).\n",
    "\n",
    "- The pollutant concentration for the same hour on the previous day (t-24).\n",
    "\n",
    "The reasoning behind this is that pollutant concentrations do not change quickly and, as previously demonstrated, have a 24-hour seasonality.\n",
    "\n",
    "**We need to be careful because we do not have values for all timestamps. To be safe, we must shift the data using pandas frequency.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data size before\n",
      "(7393, 8)\n",
      "data size after\n",
      "(7393, 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>RH</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Day</th>\n",
       "      <th>Day_of_week</th>\n",
       "      <th>Hour</th>\n",
       "      <th>is_weekend</th>\n",
       "      <th>CO_sensor_lag_1</th>\n",
       "      <th>RH_lag_1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>1224.0</td>\n",
       "      <td>56.5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1215.0</td>\n",
       "      <td>59.2</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1224.0</td>\n",
       "      <td>56.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1115.0</td>\n",
       "      <td>62.4</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1215.0</td>\n",
       "      <td>59.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1124.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1115.0</td>\n",
       "      <td>62.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1028.0</td>\n",
       "      <td>65.3</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1124.0</td>\n",
       "      <td>65.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor    RH  Month  Week  Day  Day_of_week  Hour  \\\n",
       "Date_Time                                                                   \n",
       "2004-04-04 00:00:00     1224.0  56.5      4    14    4            6     0   \n",
       "2004-04-04 01:00:00     1215.0  59.2      4    14    4            6     1   \n",
       "2004-04-04 02:00:00     1115.0  62.4      4    14    4            6     2   \n",
       "2004-04-04 03:00:00     1124.0  65.0      4    14    4            6     3   \n",
       "2004-04-04 04:00:00     1028.0  65.3      4    14    4            6     4   \n",
       "\n",
       "                     is_weekend  CO_sensor_lag_1  RH_lag_1  \n",
       "Date_Time                                                   \n",
       "2004-04-04 00:00:00           1              NaN       NaN  \n",
       "2004-04-04 01:00:00           1           1224.0      56.5  \n",
       "2004-04-04 02:00:00           1           1215.0      59.2  \n",
       "2004-04-04 03:00:00           1           1115.0      62.4  \n",
       "2004-04-04 04:00:00           1           1124.0      65.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here, I show how to move the variables forward by 1 hr,\n",
    "# so that the pollutant concentration from the previous\n",
    "# hour (t-1) is aligned with the current hour (t),\n",
    "# which is the forecasting point.\n",
    "\n",
    "# raw time series\n",
    "variables = [\"CO_sensor\", \"RH\"]\n",
    "\n",
    "# Shift the data forward 1 Hr.\n",
    "tmp = data[variables].shift(freq=\"1H\")\n",
    "\n",
    "# Names for the new variables.\n",
    "tmp.columns = [v + \"_lag_1\" for v in variables]\n",
    "\n",
    "# Add the variables to the original data.\n",
    "print(\"data size before\")\n",
    "print(data.shape)\n",
    "\n",
    "data = data.merge(tmp, left_index=True, right_index=True, how=\"left\")\n",
    "\n",
    "print(\"data size after\")\n",
    "print(data.shape)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>CO_sensor_lag_1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>1224.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1215.0</td>\n",
       "      <td>1224.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1115.0</td>\n",
       "      <td>1215.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1124.0</td>\n",
       "      <td>1115.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1028.0</td>\n",
       "      <td>1124.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor  CO_sensor_lag_1\n",
       "Date_Time                                      \n",
       "2004-04-04 00:00:00     1224.0              NaN\n",
       "2004-04-04 01:00:00     1215.0           1224.0\n",
       "2004-04-04 02:00:00     1115.0           1215.0\n",
       "2004-04-04 03:00:00     1124.0           1115.0\n",
       "2004-04-04 04:00:00     1028.0           1124.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[[\"CO_sensor\", \"CO_sensor_lag_1\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see for example that 1224 is now moved forward to the next t."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CO_sensor           0\n",
       "RH                  0\n",
       "Month               0\n",
       "Week                0\n",
       "Day                 0\n",
       "Day_of_week         0\n",
       "Hour                0\n",
       "is_weekend          0\n",
       "CO_sensor_lag_1    27\n",
       "RH_lag_1           27\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In this procedure, we introduced missing\n",
    "# data whenever there was no data available in\n",
    "# the previous hour.\n",
    "\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our timestamps are not equidistant. This means that not every row has information from the previous hour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data size before\n",
      "(7393, 10)\n",
      "data size after\n",
      "(7393, 12)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>CO_sensor_lag_24</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>1224.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1215.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1115.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1124.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1028.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 05:00:00</th>\n",
       "      <td>1010.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 06:00:00</th>\n",
       "      <td>1074.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 07:00:00</th>\n",
       "      <td>1034.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 08:00:00</th>\n",
       "      <td>1130.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 09:00:00</th>\n",
       "      <td>1275.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 10:00:00</th>\n",
       "      <td>1324.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 11:00:00</th>\n",
       "      <td>1268.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 12:00:00</th>\n",
       "      <td>1272.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 13:00:00</th>\n",
       "      <td>1160.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 14:00:00</th>\n",
       "      <td>1136.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 15:00:00</th>\n",
       "      <td>1296.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 16:00:00</th>\n",
       "      <td>1345.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 17:00:00</th>\n",
       "      <td>1296.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 18:00:00</th>\n",
       "      <td>1258.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 19:00:00</th>\n",
       "      <td>1420.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 20:00:00</th>\n",
       "      <td>1366.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 21:00:00</th>\n",
       "      <td>1113.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 22:00:00</th>\n",
       "      <td>1196.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 23:00:00</th>\n",
       "      <td>1188.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 00:00:00</th>\n",
       "      <td>1065.0</td>\n",
       "      <td>1224.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor  CO_sensor_lag_24\n",
       "Date_Time                                       \n",
       "2004-04-04 00:00:00     1224.0               NaN\n",
       "2004-04-04 01:00:00     1215.0               NaN\n",
       "2004-04-04 02:00:00     1115.0               NaN\n",
       "2004-04-04 03:00:00     1124.0               NaN\n",
       "2004-04-04 04:00:00     1028.0               NaN\n",
       "2004-04-04 05:00:00     1010.0               NaN\n",
       "2004-04-04 06:00:00     1074.0               NaN\n",
       "2004-04-04 07:00:00     1034.0               NaN\n",
       "2004-04-04 08:00:00     1130.0               NaN\n",
       "2004-04-04 09:00:00     1275.0               NaN\n",
       "2004-04-04 10:00:00     1324.0               NaN\n",
       "2004-04-04 11:00:00     1268.0               NaN\n",
       "2004-04-04 12:00:00     1272.0               NaN\n",
       "2004-04-04 13:00:00     1160.0               NaN\n",
       "2004-04-04 14:00:00     1136.0               NaN\n",
       "2004-04-04 15:00:00     1296.0               NaN\n",
       "2004-04-04 16:00:00     1345.0               NaN\n",
       "2004-04-04 17:00:00     1296.0               NaN\n",
       "2004-04-04 18:00:00     1258.0               NaN\n",
       "2004-04-04 19:00:00     1420.0               NaN\n",
       "2004-04-04 20:00:00     1366.0               NaN\n",
       "2004-04-04 21:00:00     1113.0               NaN\n",
       "2004-04-04 22:00:00     1196.0               NaN\n",
       "2004-04-04 23:00:00     1188.0               NaN\n",
       "2004-04-05 00:00:00     1065.0            1224.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we repeat the exercise, but this time\n",
    "# the values are moved forward 24 hours.\n",
    "\n",
    "# Move forward 24 hrs.\n",
    "tmp = data[variables].shift(freq=\"24H\")\n",
    "\n",
    "# Rename the variables.\n",
    "tmp.columns = [v + \"_lag_24\" for v in variables]\n",
    "\n",
    "# Add the features to the original data.\n",
    "print(\"data size before\")\n",
    "print(data.shape)\n",
    "\n",
    "data = data.merge(tmp, left_index=True, right_index=True, how=\"left\")\n",
    "\n",
    "print(\"data size after\")\n",
    "print(data.shape)\n",
    "\n",
    "data[[\"CO_sensor\", \"CO_sensor_lag_24\"]].head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how 1224, which is the value corresponding to April 4 at midnight, is now located on April 5th at midnight.\n",
    "\n",
    "We have NA for all previous rows because there is no information about the pollutant concentration 24 hours before for those rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CO_sensor             0\n",
       "RH                    0\n",
       "Month                 0\n",
       "Week                  0\n",
       "Day                   0\n",
       "Day_of_week           0\n",
       "Hour                  0\n",
       "is_weekend            0\n",
       "CO_sensor_lag_1      27\n",
       "RH_lag_1             27\n",
       "CO_sensor_lag_24    461\n",
       "RH_lag_24           461\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# In this procedure, we introduced missing\n",
    "# data whenever there was no data available in\n",
    "# the previous 24 hours.\n",
    "\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Window features\n",
    "\n",
    "Window features are mathematical computations of the features' values over a pre-defined time window, prior to the time we want to forecast.\n",
    "\n",
    "<img src='../images/window_features.png' width=\"600\" height=\"600\">\n",
    "\n",
    "For the demonstration, I will take the average of the previous 3 values of the TS to predict the current value. \n",
    "\n",
    "We first need to calculate the average of the 3 previous values, and then move that value forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor_window</th>\n",
       "      <th>RH_window</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1224.000000</td>\n",
       "      <td>56.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1219.500000</td>\n",
       "      <td>57.850000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1184.666667</td>\n",
       "      <td>59.366667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1151.333333</td>\n",
       "      <td>62.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 05:00:00</th>\n",
       "      <td>1089.000000</td>\n",
       "      <td>64.233333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 06:00:00</th>\n",
       "      <td>1054.000000</td>\n",
       "      <td>65.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 07:00:00</th>\n",
       "      <td>1037.333333</td>\n",
       "      <td>66.966667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 08:00:00</th>\n",
       "      <td>1039.333333</td>\n",
       "      <td>66.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 09:00:00</th>\n",
       "      <td>1079.333333</td>\n",
       "      <td>64.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 10:00:00</th>\n",
       "      <td>1146.333333</td>\n",
       "      <td>57.866667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor_window  RH_window\n",
       "Date_Time                                       \n",
       "2004-04-04 01:00:00       1224.000000  56.500000\n",
       "2004-04-04 02:00:00       1219.500000  57.850000\n",
       "2004-04-04 03:00:00       1184.666667  59.366667\n",
       "2004-04-04 04:00:00       1151.333333  62.200000\n",
       "2004-04-04 05:00:00       1089.000000  64.233333\n",
       "2004-04-04 06:00:00       1054.000000  65.600000\n",
       "2004-04-04 07:00:00       1037.333333  66.966667\n",
       "2004-04-04 08:00:00       1039.333333  66.800000\n",
       "2004-04-04 09:00:00       1079.333333  64.300000\n",
       "2004-04-04 10:00:00       1146.333333  57.866667"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use the mean of the 3 previous hours as input variables.\n",
    "\n",
    "tmp = (\n",
    "    data[variables]\n",
    "    .rolling(window=\"3H\")\n",
    "    .mean()  # Average the last 3 hr values.\n",
    "    .shift(freq=\"1H\")  # Move the average 1 hour forward\n",
    ")\n",
    "\n",
    "# Rename the columns\n",
    "tmp.columns = [v + \"_window\" for v in variables]\n",
    "\n",
    "\n",
    "# view of the result\n",
    "tmp.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data size before\n",
      "(7393, 12)\n",
      "data size after\n",
      "(7393, 14)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>CO_sensor_window</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>1224.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>1215.0</td>\n",
       "      <td>1224.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>1115.0</td>\n",
       "      <td>1219.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>1124.0</td>\n",
       "      <td>1184.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>1028.0</td>\n",
       "      <td>1151.333333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor  CO_sensor_window\n",
       "Date_Time                                       \n",
       "2004-04-04 00:00:00     1224.0               NaN\n",
       "2004-04-04 01:00:00     1215.0       1224.000000\n",
       "2004-04-04 02:00:00     1115.0       1219.500000\n",
       "2004-04-04 03:00:00     1124.0       1184.666667\n",
       "2004-04-04 04:00:00     1028.0       1151.333333"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Join the new variables to the original data.\n",
    "print(\"data size before\")\n",
    "print(data.shape)\n",
    "\n",
    "data = data.merge(tmp, left_index=True, right_index=True, how=\"left\")\n",
    "\n",
    "print(\"data size after\")\n",
    "print(data.shape)\n",
    "\n",
    "data[[\"CO_sensor\", \"CO_sensor_window\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we do some manual calculations to convince ourselves of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1219.5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1215 + 1224) / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1184.6666666666667"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1115 + 1215 + 1224) / 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important:** Notice how the average of the previous three hours was moved forward an hour to time t, the time we want to forecast."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Periodic features\n",
    "\n",
    "Some features are periodic. For example, hours, months, and days.\n",
    "\n",
    "We can encode those periodic features using a sine and cosine transformation with the feature's period. This will cause the values of the features that are far apart to come closer. For example, December (12) is closer to January (1) than June (6). This relationship is not captured by the numerical representation of these features. But we could change it, if we transformed these variables with sine and cosine.\n",
    "\n",
    "We will discuss this technique later on in the course. For now, let's create these features automatically with the open source library Feature-engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create features that capture the cyclical representation.\n",
    "\n",
    "cyclical = CyclicalFeatures(\n",
    "    variables=[\"Month\", \"Hour\"],  # The features we want to transform.\n",
    "    drop_original=False,  # Whether to drop the original features.\n",
    ")\n",
    "\n",
    "data = cyclical.fit_transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month_sin</th>\n",
       "      <th>Month_cos</th>\n",
       "      <th>Hour_sin</th>\n",
       "      <th>Hour_cos</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-04 00:00:00</th>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 01:00:00</th>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.269797</td>\n",
       "      <td>0.962917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 02:00:00</th>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.519584</td>\n",
       "      <td>0.854419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 03:00:00</th>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.730836</td>\n",
       "      <td>0.682553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-04 04:00:00</th>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.887885</td>\n",
       "      <td>0.460065</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Month_sin  Month_cos  Hour_sin  Hour_cos\n",
       "Date_Time                                                    \n",
       "2004-04-04 00:00:00   0.866025       -0.5  0.000000  1.000000\n",
       "2004-04-04 01:00:00   0.866025       -0.5  0.269797  0.962917\n",
       "2004-04-04 02:00:00   0.866025       -0.5  0.519584  0.854419\n",
       "2004-04-04 03:00:00   0.866025       -0.5  0.730836  0.682553\n",
       "2004-04-04 04:00:00   0.866025       -0.5  0.887885  0.460065"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cyclical_vars = [var for var in data.columns if \"sin\" in var or \"cos\" in var]\n",
    "\n",
    "data[cyclical_vars].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the newly created features at the end of the dataframe.\n",
    "\n",
    "## Drop missing data\n",
    "\n",
    "When creating lag and window features, we introduced missing data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CO_sensor           0.000000\n",
       "RH                  0.000000\n",
       "Month               0.000000\n",
       "Week                0.000000\n",
       "Day                 0.000000\n",
       "Day_of_week         0.000000\n",
       "Hour                0.000000\n",
       "is_weekend          0.000000\n",
       "CO_sensor_lag_1     0.003652\n",
       "RH_lag_1            0.003652\n",
       "CO_sensor_lag_24    0.062356\n",
       "RH_lag_24           0.062356\n",
       "CO_sensor_window    0.003652\n",
       "RH_window           0.003652\n",
       "Month_sin           0.000000\n",
       "Month_cos           0.000000\n",
       "Hour_sin            0.000000\n",
       "Hour_cos            0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Determine fraction of missing data.\n",
    "\n",
    "data.isnull().sum() / len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imputation\n",
    "\n",
    "There is not a lot of data missing, so I will just remove those observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data size before\n",
      "(7393, 18)\n",
      "data size after\n",
      "(6922, 18)\n"
     ]
    }
   ],
   "source": [
    "print(\"data size before\")\n",
    "print(data.shape)\n",
    "\n",
    "data.dropna(inplace=True)\n",
    "\n",
    "print(\"data size after\")\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save preprocessed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>RH</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Day</th>\n",
       "      <th>Day_of_week</th>\n",
       "      <th>Hour</th>\n",
       "      <th>is_weekend</th>\n",
       "      <th>CO_sensor_lag_1</th>\n",
       "      <th>RH_lag_1</th>\n",
       "      <th>CO_sensor_lag_24</th>\n",
       "      <th>RH_lag_24</th>\n",
       "      <th>CO_sensor_window</th>\n",
       "      <th>RH_window</th>\n",
       "      <th>Month_sin</th>\n",
       "      <th>Month_cos</th>\n",
       "      <th>Hour_sin</th>\n",
       "      <th>Hour_cos</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-05 00:00:00</th>\n",
       "      <td>1065.0</td>\n",
       "      <td>65.8</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1188.0</td>\n",
       "      <td>60.8</td>\n",
       "      <td>1224.0</td>\n",
       "      <td>56.5</td>\n",
       "      <td>1165.666667</td>\n",
       "      <td>58.566667</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 01:00:00</th>\n",
       "      <td>999.0</td>\n",
       "      <td>79.2</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1065.0</td>\n",
       "      <td>65.8</td>\n",
       "      <td>1215.0</td>\n",
       "      <td>59.2</td>\n",
       "      <td>1149.666667</td>\n",
       "      <td>61.800000</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.269797</td>\n",
       "      <td>0.962917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 02:00:00</th>\n",
       "      <td>911.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>999.0</td>\n",
       "      <td>79.2</td>\n",
       "      <td>1115.0</td>\n",
       "      <td>62.4</td>\n",
       "      <td>1084.000000</td>\n",
       "      <td>68.600000</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.519584</td>\n",
       "      <td>0.854419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 03:00:00</th>\n",
       "      <td>873.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>911.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>1124.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>991.666667</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.730836</td>\n",
       "      <td>0.682553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 04:00:00</th>\n",
       "      <td>881.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>873.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>1028.0</td>\n",
       "      <td>65.3</td>\n",
       "      <td>927.666667</td>\n",
       "      <td>80.066667</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.887885</td>\n",
       "      <td>0.460065</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor    RH  Month  Week  Day  Day_of_week  Hour  \\\n",
       "Date_Time                                                                   \n",
       "2004-04-05 00:00:00     1065.0  65.8      4    15    5            0     0   \n",
       "2004-04-05 01:00:00      999.0  79.2      4    15    5            0     1   \n",
       "2004-04-05 02:00:00      911.0  80.0      4    15    5            0     2   \n",
       "2004-04-05 03:00:00      873.0  81.0      4    15    5            0     3   \n",
       "2004-04-05 04:00:00      881.0  81.0      4    15    5            0     4   \n",
       "\n",
       "                     is_weekend  CO_sensor_lag_1  RH_lag_1  CO_sensor_lag_24  \\\n",
       "Date_Time                                                                      \n",
       "2004-04-05 00:00:00           0           1188.0      60.8            1224.0   \n",
       "2004-04-05 01:00:00           0           1065.0      65.8            1215.0   \n",
       "2004-04-05 02:00:00           0            999.0      79.2            1115.0   \n",
       "2004-04-05 03:00:00           0            911.0      80.0            1124.0   \n",
       "2004-04-05 04:00:00           0            873.0      81.0            1028.0   \n",
       "\n",
       "                     RH_lag_24  CO_sensor_window  RH_window  Month_sin  \\\n",
       "Date_Time                                                                \n",
       "2004-04-05 00:00:00       56.5       1165.666667  58.566667   0.866025   \n",
       "2004-04-05 01:00:00       59.2       1149.666667  61.800000   0.866025   \n",
       "2004-04-05 02:00:00       62.4       1084.000000  68.600000   0.866025   \n",
       "2004-04-05 03:00:00       65.0        991.666667  75.000000   0.866025   \n",
       "2004-04-05 04:00:00       65.3        927.666667  80.066667   0.866025   \n",
       "\n",
       "                     Month_cos  Hour_sin  Hour_cos  \n",
       "Date_Time                                           \n",
       "2004-04-05 00:00:00       -0.5  0.000000  1.000000  \n",
       "2004-04-05 01:00:00       -0.5  0.269797  0.962917  \n",
       "2004-04-05 02:00:00       -0.5  0.519584  0.854419  \n",
       "2004-04-05 03:00:00       -0.5  0.730836  0.682553  \n",
       "2004-04-05 04:00:00       -0.5  0.887885  0.460065  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CO_sensor</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "      <th>Day</th>\n",
       "      <th>Day_of_week</th>\n",
       "      <th>Hour</th>\n",
       "      <th>is_weekend</th>\n",
       "      <th>CO_sensor_lag_1</th>\n",
       "      <th>RH_lag_1</th>\n",
       "      <th>CO_sensor_lag_24</th>\n",
       "      <th>RH_lag_24</th>\n",
       "      <th>CO_sensor_window</th>\n",
       "      <th>RH_window</th>\n",
       "      <th>Month_sin</th>\n",
       "      <th>Month_cos</th>\n",
       "      <th>Hour_sin</th>\n",
       "      <th>Hour_cos</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date_Time</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2004-04-05 00:00:00</th>\n",
       "      <td>1065.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1188.0</td>\n",
       "      <td>60.8</td>\n",
       "      <td>1224.0</td>\n",
       "      <td>56.5</td>\n",
       "      <td>1165.666667</td>\n",
       "      <td>58.566667</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 01:00:00</th>\n",
       "      <td>999.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1065.0</td>\n",
       "      <td>65.8</td>\n",
       "      <td>1215.0</td>\n",
       "      <td>59.2</td>\n",
       "      <td>1149.666667</td>\n",
       "      <td>61.800000</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.269797</td>\n",
       "      <td>0.962917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 02:00:00</th>\n",
       "      <td>911.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>999.0</td>\n",
       "      <td>79.2</td>\n",
       "      <td>1115.0</td>\n",
       "      <td>62.4</td>\n",
       "      <td>1084.000000</td>\n",
       "      <td>68.600000</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.519584</td>\n",
       "      <td>0.854419</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 03:00:00</th>\n",
       "      <td>873.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>911.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>1124.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>991.666667</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.730836</td>\n",
       "      <td>0.682553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004-04-05 04:00:00</th>\n",
       "      <td>881.0</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>873.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>1028.0</td>\n",
       "      <td>65.3</td>\n",
       "      <td>927.666667</td>\n",
       "      <td>80.066667</td>\n",
       "      <td>0.866025</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.887885</td>\n",
       "      <td>0.460065</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     CO_sensor  Month  Week  Day  Day_of_week  Hour  \\\n",
       "Date_Time                                                             \n",
       "2004-04-05 00:00:00     1065.0      4    15    5            0     0   \n",
       "2004-04-05 01:00:00      999.0      4    15    5            0     1   \n",
       "2004-04-05 02:00:00      911.0      4    15    5            0     2   \n",
       "2004-04-05 03:00:00      873.0      4    15    5            0     3   \n",
       "2004-04-05 04:00:00      881.0      4    15    5            0     4   \n",
       "\n",
       "                     is_weekend  CO_sensor_lag_1  RH_lag_1  CO_sensor_lag_24  \\\n",
       "Date_Time                                                                      \n",
       "2004-04-05 00:00:00           0           1188.0      60.8            1224.0   \n",
       "2004-04-05 01:00:00           0           1065.0      65.8            1215.0   \n",
       "2004-04-05 02:00:00           0            999.0      79.2            1115.0   \n",
       "2004-04-05 03:00:00           0            911.0      80.0            1124.0   \n",
       "2004-04-05 04:00:00           0            873.0      81.0            1028.0   \n",
       "\n",
       "                     RH_lag_24  CO_sensor_window  RH_window  Month_sin  \\\n",
       "Date_Time                                                                \n",
       "2004-04-05 00:00:00       56.5       1165.666667  58.566667   0.866025   \n",
       "2004-04-05 01:00:00       59.2       1149.666667  61.800000   0.866025   \n",
       "2004-04-05 02:00:00       62.4       1084.000000  68.600000   0.866025   \n",
       "2004-04-05 03:00:00       65.0        991.666667  75.000000   0.866025   \n",
       "2004-04-05 04:00:00       65.3        927.666667  80.066667   0.866025   \n",
       "\n",
       "                     Month_cos  Hour_sin  Hour_cos  \n",
       "Date_Time                                           \n",
       "2004-04-05 00:00:00       -0.5  0.000000  1.000000  \n",
       "2004-04-05 01:00:00       -0.5  0.269797  0.962917  \n",
       "2004-04-05 02:00:00       -0.5  0.519584  0.854419  \n",
       "2004-04-05 03:00:00       -0.5  0.730836  0.682553  \n",
       "2004-04-05 04:00:00       -0.5  0.887885  0.460065  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop Relative humidity raw (we do not know its\n",
    "# values at time of forecast)\n",
    "\n",
    "data.drop(\"RH\", inplace=True, axis=1)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# store new dataset\n",
    "\n",
    "data.to_csv(\"air_qual_preprocessed.csv\", index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fsml",
   "language": "python",
   "name": "fsml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
